Http oracle timeout by omursahin · Pull Request #1608 · WebFuzzing/EvoMaster

omursahin · 2026-06-23T19:33:04Z

Python test case

    # Calls:
    # (null) GET:/api/timeout/slow/{id}
    # Found 1 potential fault of type-code 921
    @timeout_decorator.timeout(60)
    def test_0_get_on_slow_timeout(self):
        
        # Fault921. Timeout.
        headers = {}
        headers['Accept'] = "*/*"
        with self.assertRaises(Exception):
            requests \
                    .get(self.baseUrlOfSut + "/api/timeout/slow/683",
                        headers=headers, timeout=EM_HTTP_TIMEOUT, verify=False)

JS test case

/**
* Calls:
* (null) GET:/api/timeout/slow/{id}
* Found 1 potential fault of type-code 921
*/
test("test_0_GetOnSlowTimeout", async () => {
    
    // Fault921. Timeout.
    await expect(superagent
            .get(baseUrlOfSut + "/api/timeout/slow/683")
            .timeout({response: EM_HTTP_TIMEOUT_MS, deadline: EM_HTTP_TIMEOUT_MS}).set('Accept', "*/*")
            .ok(res => res.status)).rejects.toThrow();
});

Kotlin:

        @BeforeAll
        @JvmStatic
        fun initClass() {
            RestAssured.enableLoggingOfRequestAndResponseIfValidationFails()
            RestAssured.useRelaxedHTTPSValidation()
            RestAssured.urlEncodingEnabled = false
            RestAssured.config = RestAssured.config()
                .jsonConfig(JsonConfig.jsonConfig().numberReturnType(JsonPathConfig.NumberReturnType.DOUBLE))
                .redirect(redirectConfig().followRedirects(false))
                .httpClient(HttpClientConfig.httpClientConfig()
                    .setParam("http.socket.timeout", 2000)
                    .setParam("http.connection.timeout", 2000))
                .jsonConfig(JsonConfig.jsonConfig().numberReturnType(JsonPathConfig.NumberReturnType.DOUBLE))
                .redirect(redirectConfig().followRedirects(false))
                .encoderConfig(EncoderConfig.encoderConfig().encodeContentTypeAs("application/octet-stream", ContentType.TEXT))
        }

    /**
    * Calls:
    * (null) GET:/api/timeout/slow/{id}
    * Found 1 potential fault of type-code 921
    */
    @Test @Timeout(60)
    fun test_0_getOnSlowTimeout()  {
        
        try{
            // Fault921. Timeout.
            given().accept("*/*")
                    .get("${baseUrlOfSut}/api/timeout/slow/683")
            fail("Expected a timeout");
        } catch(e: Exception){
        }
    }

Java

    @BeforeClass
    public static void initClass() {
        RestAssured.enableLoggingOfRequestAndResponseIfValidationFails();
        RestAssured.useRelaxedHTTPSValidation();
        RestAssured.urlEncodingEnabled = false;
        RestAssured.config = RestAssured.config()
            .jsonConfig(JsonConfig.jsonConfig().numberReturnType(JsonPathConfig.NumberReturnType.DOUBLE))
            .redirect(redirectConfig().followRedirects(false))
            .httpClient(HttpClientConfig.httpClientConfig()
                .setParam("http.socket.timeout", 2000)
                .setParam("http.connection.timeout", 2000))
            .jsonConfig(JsonConfig.jsonConfig().numberReturnType(JsonPathConfig.NumberReturnType.DOUBLE))
            .redirect(redirectConfig().followRedirects(false))
            .encoderConfig(EncoderConfig.encoderConfig().encodeContentTypeAs("application/octet-stream", ContentType.TEXT));
    }
        /**
    * Calls:
    * (null) GET:/api/timeout/slow/{id}
    * Found 1 potential fault of type-code 921
    */
    @Test(timeout = 60000)
    public void test_0_getOnSlowTimeout() throws Exception {
        
        try{
            // Fault921. Timeout.
            given().accept("*/*")
                    .get(baseUrlOfSut + "/api/timeout/slow/683");
            fail("Expected a timeout");
        } catch(Exception e){
        }
    }

arcuri82 · 2026-07-03T12:13:33Z

                //in SuperAgent, verb must be first
                handleVerbEndpoint(baseUrlOfSut, call, lines)
+                //client timeout, same source as fuzzing tcpTimeoutMs
+                lines.add(".timeout({response: EM_HTTP_TIMEOUT_MS, deadline: EM_HTTP_TIMEOUT_MS})")


this EM_HTTP_TIMEOUT_MS is referring to a variable declared elsewhere? in such case, the name of the variable should be put in a shared constant

arcuri82 · 2026-07-03T12:16:25Z

+
+        // message for the assertion that flags a missing expected timeout (Java/Kotlin/C#)
+        // JS uses await expect(...).rejects.toThrow() and Python uses with self.assertRaises(...)
+        private const val EXPECTED_TIMEOUT_MSG = "Expected a timeout"


use JavaDoc style /** */ for comments on fields and methos

arcuri82 · 2026-07-03T12:18:18Z

            lines.add("process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';")
            lines.add("const superagent = require(\"superagent\");")
+            // HTTP client timeout (ms)
+            lines.add("const EM_HTTP_TIMEOUT_MS = ${config.tcpTimeoutMs};")


this name EM_HTTP_TIMEOUT_MS should be in a shared constant, as re-used in few places

arcuri82 · 2026-07-03T12:18:58Z

            }
            lines.add("from $pythonUtilsFilenameNoExtension import *")
+            // HTTP client timeout (seconds)
+            lines.add("EM_HTTP_TIMEOUT = ${config.tcpTimeoutMs / 1000.0}")


shared constant

arcuri82 · 2026-07-03T12:20:05Z

            analyzeHttpSemantics(individual, actionResults, fv)
        }

+        if(config.blackBox && config.isEnabledFaultCategory(ExperimentalFaultCategory.HTTP_TIMEOUT)){


add comment specifying why we are skipping for white-box testing

arcuri82 · 2026-07-03T12:21:22Z

            add(".get(self.baseUrlOfSut + \"/\",")
            indent()
-            add("headers=headers, verify=False)")
+            add("headers=headers, timeout=EM_HTTP_TIMEOUT, verify=False)")


otherwise, otherwise if we change name in future, we will break lot of places

arcuri82 · 2026-07-03T12:22:05Z

                res_0 = requests \
                        .get(self.baseUrlOfSut + "/foo",
-                            headers=headers, verify=False)
+                            headers=headers, timeout=EM_HTTP_TIMEOUT, verify=False)


constant. also, we need a way to make it easier and do not confuse when we us MS or S version

arcuri82 · 2026-07-03T12:25:50Z

+        ) { args: MutableList<String> ->
+
+            setOption(args, "useExperimentalOracles", "true")
+            setOption(args, "tcpTimeoutMs", "2000")


shouldn't this test fail because by default we have httpOracles being false?

@omursahin this is not addressing my concern. if i take this test, and put that httpOracles to false, the test case still pass. if HTTP Oracles are not activated, the timeout oracle should not be evaluated, and, as such, this test case should fail. the fact that it still pass is a bug

arcuri82 · 2026-07-06T07:11:22Z

+        ) { args: MutableList<String> ->
+
+            setOption(args, "useExperimentalOracles", "true")
+            setOption(args, "tcpTimeoutMs", "2000")


@omursahin this is not addressing my concern. if i take this test, and put that httpOracles to false, the test case still pass. if HTTP Oracles are not activated, the timeout oracle should not be evaluated, and, as such, this test case should fail. the fact that it still pass is a bug

omursahin and others added 5 commits June 23, 2026 22:32

timeout

89ec54a

setting timeout in tests

e7cabb8

http timeout write test case

6b6353f

add timeout param

6c23581

Merge branch 'master' into http-oracle-timeout

e637561

omursahin requested a review from arcuri82 July 1, 2026 10:51

arcuri82 requested changes Jul 3, 2026

View reviewed changes

omursahin and others added 5 commits July 3, 2026 18:03

enable http oracles

c5a764a

Merge branch 'master' into http-oracle-timeout

f5c01e2

http semantics for bb

811208f

fitness binding in bb

3bb0ca4

variable name constant

8d8b308

omursahin requested a review from arcuri82 July 5, 2026 13:17

arcuri82 requested changes Jul 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Http oracle timeout#1608

Http oracle timeout#1608
omursahin wants to merge 10 commits into
masterfrom
http-oracle-timeout

omursahin commented Jun 23, 2026 •

edited

Loading

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 3, 2026

Uh oh!

arcuri82 Jul 6, 2026

Uh oh!

arcuri82 Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

omursahin commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

omursahin commented Jun 23, 2026 •

edited

Loading